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BACKGROUND OF THE INVENTION 

Field of the Invention 

[0001] This invention relates to the technologies of online and off-line shopping and 
e-commerce, and especially to the graphical technologies for representing commerce 
5 facilities, organization, and navigation of shopping centers using communications 
terminal devices such as personal digital assistants, handheld personal computers, 
Internet appliances, and web-enabled telephones. 



Description of the Related Art 

[0002] In today's competitive commercial and retail environment, owners and 
10 employees of physical ''bricks and mortar" stores often view web-based commerce as 
competition. For example, a local bookstore owner recognizes such online proprietors 
as Amazon.com as a direct competitor. In some instances, retail franchise owners 
may view the online web site of the same franchise name, albeit operated by the master 
franchiser, as even more direct competition because there is a high level of overlap in 
1 5 inventory as well as identical name brand recognition. 

[0003] Many online shopping "malls", or "cybermalls", have been developed as 
web-based purchasing has become culturally acceptable to consumers and as online 
purchasing security concerns have been addressed. However, these online shopping 
malls are typically little more than a group of hyperlinked web sites or portions of web 
20 sites, accessible through a common "home" page. Tiiming to FIGURE 1 , the 
well-known arrangement of web browser computers (1) and web servers (5) 
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interconnected by the Internet or World Wide Web (3) or intranets (6). Typically, the 
browser computer (1) comprises a personal computer running a web browser software 
such as Netscape's Navigator, using a protocol such as Transmission Control 
Protocol/Internet Protocol (TCP/IP) running over a dial-up modem connect, digital 
5 subscriber line (DSL), cable modem, or the like. The web server (5) typically consists 
of a web platform, such as IBM's Websphere product, and communicates to browser 
computers using Hyper Text Transfer Protocol (HTTP) by transmitting web objects 
including Hyper Text Markup Language Documents (HTML), graphic images (GIF, 
JPEG, etc.), audio and video clips (AVI, WAV, etc.), JAVA applets, and other 
10 common types of content objects. Hyperlinking for automatically addressing these 
types of web objects is well known in the art, and is prevalent throughout today's web 
environment. 

[0004] Cybermalls exist currently as a loose collection of store web sites, for 
example, a grouping of online shoe stores accessible by a single hop or "click" from a 

1 5 common access point. Some cybermalls are collections of store sites offering products 
with geographical relationships, such as products made only in New England, Idaho, 
or Hawaii. Many bricks-and-mortar malls provide a variety of store types, including 
some specialty stores, some department stores, and meeting places such as food courts 
and restaurants, as do many cybermalls. 

20 [0005] During the 1980's when automatic teller machines (ATM) became widely 
available for bank customers, banks found that their ability to distinquish themselves 
from competitors was subsequently reduced as their clients visited their physical 
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facilities less and less often, favoring to make most transactions with a convenient 
ATM. For example, a first bank may have estabUshed a very respectable, rehable 
image, with bank lobbies furnished in luxurious fiimiture, marble, and artwork. 
CUents who frequent this bank's lobby are given an impression that the bank is well 
5 established, and higher fees may be more acceptable. Another bank, perhaps a credit 
union, may adopt a more Spartan image, with more practical and cost effective 
furniture, such as ''cube" furniture, in order to project an image of efficiency and cost 
effectiveness. 

[0006] However, as bank chents began to conduct the vast majority of their banking 
1 0 transactions with an ATM, they visited the bank's actual facilities very seldom and in 
some cases never. So, the client's view of the bank became that of the ATM terminal 
not the bank's lobby or building. Since there is very httle difference between the 
appearances of ATM terminals, banks subsequently found it very difficult to 
distinguish their desired image fi-om their competitors images. 
1 5 [0007] So has become the problem for retailers in malls and online. While in 
physical form in a mall, a retailer may use choices of lighting, display materials and 
designs, background sounds and music, and store facade design to generate an image, 
it is very difficult to present the same image in a distinguishing manner through a web 
browser. As such, some online-only retailers, such as Amazon.com, have been able to 
20 quickly estabhsh an image comparable and competitive with bricks-and-mortar 
retailers such as Borders and Same's and Noble. 
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[0008] Presently, online marketplaces are frequently set up in one of two 
fundamental ways. Firstly, consider that stores and malls may be presented in their 
entirety as a single domain, with possible divisions between departments (e.g., men's 
wear, households, etc), as shown in FIGURE 2. A mall home page (21) may contain a 
5 group of hyperlinks to various store home pages (23, 24, and 25), which in turn 
provide hyperlinks to department pages (26, 27, 28, 29, 200, and 201). The tree 
structure of these sites are well known and are not unlike the tree structures of other, 
non-retail web sites. 

[0009] Secondly, online malls are often organized so that visiting one "mall-front" 
10 shows lists of stores of possible interest to the visitor, and often provide search 

facilities (36) based on store names or product categories, as shown in FIGURE 3. In 
this example, the web browser framer (31) which is displayed on a portion (30) of a 
web browsing device's display provides BACK and FORWARD navigation buttons 
(33, 34), and a location or address entry (32). Some online malls, as well as some 
15 online convention halls (wherein virtual convention "booths" are presented), provide a 
map-like view (37) of the virtual "layout" of the mall. This usually does not 
correspond to a real mall design, but is presented to enhance the browser's shopping 
experience. The cursor or pointer (35) may be used by the browser to select an icon, 
button, or store on the map. Again, similar to the first method described, these stores 
20 may either be organized as separate domains or accessible through the same domain. 
Again, too, it is evident that this organization of information is not unlike organization 
of information on other types of non-retail web sites. Thus, the "look and feel" of 
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visiting these types of online malls is not much different that that of visiting other types 
of web sites, and certainly does not parallel the full sensory experience of visiting a 
real, bricks-and-mortar mail Current online malls have little commonality or 
coherence to result in return visits, known as "stickiness", when compared to real 
5 shopping malls. 

[0010] Many of the carefully selected factors in real stores are lost in the online 
shopping experience. A mall operator may group certain types of stores based on a 
crossover business potential, such as placing a linen store, a bath products store, and a 
women's dress shop in close proximity to each other to target female shoppers. 

1 0 Within "cyberstores" themselves, other factors have not been translated well to the 
online experience, such as the use of background music and sounds to set a mood or 
environment to complement a store's or department's product lines. 
[0011] The related application disclosed a new method and system through which a 
mall shopper of an online shopping mall may be presented with an multimedia 

15 experience similar to a real shopping experience, including the ability to graphically 
navigate a map of a shopping mall, view images of mall structures such as store 
facades and hallways, enter stores and navigate hallways through selection of hot spots 
in images, and navigate within stores while being presented with images of the store 
interior spaces and products. During all of these processes, relevant sounds, such as 

20 background noise in the open spaces and background music within a store, are 
presented to the shopper, as well, in order to complete the pseudo-real shopping 
experience. 
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[0012] However, the system and method described were disclosed in terms of online 
web browsing, preferably as a Java applet executed by a web browser with real-time 
delivery of image, sound and data files as the shopper navigates an online mall. There 
is a growing class of Internet appliances, web-enabled telephones, web-enabled 

5 personal digital assistants and handheld computers which are capable of 

communications with a network server over a communications network, but do not 
have the processing bandwidth or memory of full-capability web browser computers. 
[0013] The comparative resource limitations of such mobile devices will likely 
continue because as memory and processor bandwidth become more affordable, each 

10 mobile device's resources will be increased accordingly, but so will each computer's 
resources. As such, there will likely be a considerable disparity between the 
resources available on a desktop or laptop computer when compared to the resources 
and capabilities of a mobile device. 

[0014] Therefore, there is a need in the art for a system and method which provides 
15 a more realistic shopping experience for virtual shopping malls and complexes using a 
web-enabled mobile device. Additionally, there is a need in the art for the new system 
and method to support online and off-line shopping modes as online "connection time" 
is typically a premium resource to be minimized by any apphcation of the device. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



[0015] The following detailed description when taken in conjunction with the figures 
presented herein provide a complete disclosure of the invention. 
5 [0016] FIGURE 1 illustrates the well-known arrangement of web browser devices 
and web servers, 

[0017] FIGURE 2 shows the typical tree-like structure or organization of online 
mall information. 

[0018] FIGURE 3 shows a typical online mall front page with a "map" of a virtual 
10 mall. 

[0019] FIGURE 4 discloses the enhanced view of an online mall front page including 
a "you are here" indicator, and an icon to select the full sensory presentation of the 
mall shopping visit. 

[0020] FIGURE 5 shows an example presentation of a visual image from a particular 
1 5 vantage point or position within a cybermall in which multiple adjacent store fronts 
may been seen. 

[0021] FIGURE 6 sets forth the common arrangement of components of mobile 
devices such as PDA's, web enabled telephones, and handheld personal computers. 
[0022] FIGURE 7 illustrates the logical process performed by the invention. 
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SUMMARY OF THE INVENTION 
[0023] The present invention provides an enhanced sensory experience coupled to 
an virtual shopping mall which creates an apparent geographical coupling between 
cyberstores within the virtual mall, and which enables store proprietors to control 
5 shopping environment factors, 

[0024] Using presentation of visual images and audible sounds relevant and 
coordinated to a shopper's "position" within the cybermall, a shopper may view 
images and hear sounds simulating a real shopping mall experience using a personal 
digital assistant, web-enabled wireless telephone, or similar mobile device having a 

1 0 graphical display. 

[0025] As a shopper "moves" through the cybermall, graphical images of mall 
hallways are presented in logical sequence showing store fronts and facades, with 
selectable areas in the images defining entry points to enter the stores. As stores are 
entered, specific images of store interiors are provided, allowing each store to control 

1 5 and generate an enviroimient within their own store. During the entire experience, 
relevant audible sounds are provided to the shopper such as general mall hallway 
sounds while in the mall, and store-specific background sounds and music while in a 
selected store. 

[0026] The navigation, image, and audio facilities are provided by an application 
20 program loaded to a mobile device to enable the invention to be utilized with a wide 
variety of mobile devices and platforms. According to the preferred embodiment, 
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images, audio, and data are downloaded and stored to riunimize online connection 
time, and to facilitate off-line shopping. 
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DETAILED DESCRIPTION OF THE INVENTION 
[0027[ The invention is provided preferably as an application program executable by 
a mobile device such as a personal digital assistant (PDA), web-enabled wireless 
telephone, handheld personal computer, or other Internet appliance, 
5 [0028] Turning to FIGURE 6, the generaUzed arrangement of such web-enabled 
mobile devices (70) is shown. The mobile device includes a CPU or microprocessorr 
(74), system memory (75) such as RAM and ROM, and a set of user interface devices 
(73) including a graphical display such as a color LCD panel. Other user interface 
devices commonly provided on this mobile devices include a touch-screen input, 

10 keyboard, pointing device, speakers or annunciators, microphone and camera. 

[0029] The mobile device (70) is also provided with an operating system and set of 
library functions (703). The library functions may include a communications protocol 
stack such as TCP/IP, electronic mail functions such as a Simple Mail Transfer 
Protocol (SMTP) suite, and user interface functions such as handwriting recognizors. 

15 A suitable communciations network interface (77) to a communications network (71) 
is also preferrably provided in the mobile device, such as a dial-up modem, wireless 
modem, or InfraRed communications port (IRDa). Device drivers (78) allow 
abstraction between the library functions and the application programs (702). 
[0030] These mobile devices are typically provided with a suite of resident or 

20 pre-loaded appUcations such as phone books, "to do" list managers, calendars, email 
browsers, etc.. Other applications may be installed by downloading from a 
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coxmrLunications network server, or by installation from a memory device such as a 
FlashROM card, PCMCIA card, etc. 

[0031] This general representation of mobile devices is representative of a wide 
variety of available products, from PDA's such as the Palm Pilot [TM], to web-enabled 
5 wireless telephones such as Motorola's StarTac[TM], to Internet appliances such as 
Cidco's MailStation[TM], According to the preferred embodiment, the invention is 
realized in combination with a Palm model m505 PDA, which has a high-resolution 
16-bit color LCD display, PaknOS [TM] operating system, infrared communications 
port, 8 MB of system memory, touchscreen input, and a FlashROM card port. Palm 

1 0 provides two options for interfacing to a communications network through connection 
to a mobile telephone, such as a PCS handset, or using an integral wireless modem. 
By equipping the PDA with either communication option, application programs may 
access the Internet and send/receive electronic mail using PalmOS library functions. 
[0032] While the Palm PDA is selected as the platform of the preferred embodiment 

15 due to its open operating system and superior display characteristics, it will be 
recognized by those skilled in the art that realization of the invention on any other 
suitable platform may be made without departing from the spirit and scope of the 
invention. 

[0033] The remainder of the disclosure given herein is made in terms of the method 
20 implemented in an apphcation program targeted for the mobile device. According to 
the preferred embodiment, the method set forth is embodied in a PalmOS-compatible 
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application program using the Palm software developers kit (SDK). Several suitable 
C and C++ compilers are available for use in compiling the application. 
[0034] Tuming to FIGURE 4, the enhanced mall front page (31) provided by the 
invention is shown. This graphic image is displayed on a portion (30) of the mobile 
5 device's graphical display. In this view, the mall map (37) is enhanced to include a 
shopper position indicator (38), and preferably a geographic orientation icon (300). 
[0035] Based upon a current coordinate or position of the "shopper" within the mall, 
maintained by the application program, the shopper may see his or her position within 
the mall floorplan. Preferably, the shopper may "chck and drag" the position indicator 
10 (38) using the pointer or touchscreen stylus (37) to move to another position within 
the mall, or go directly to another position within the mall floorplan by double cHcking 
anywhere in the floorplan. 

[0036] Further, the mall front page is enhanced to include a "walk the mall" button 
or icon (39), which invokes the multimedia experience of the invention. This could 

1 5 alternatively be invoked by the double clicking action previously described. 

[0037] FIGURE 5 shows an example presentation of the visual image from a given 
position within a mall floor plan, including store facades, and the position indicator 
(38). The position indicator (38) may be provided with arrows or pointers showing 
possible directions of movement. A view indicator (62) also may be provided to 

20 allow the shopper greater understanding of which direction he or she is "facing" in the 
cybermall, as well as a miniaturized mall floor plan (63) for more efficient navigation 
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of the cybermall A button or icon (60) to return to the mall front page is also 
preferably provided. 

[0038] From this view, the user or shopper may click and drag the position indicator 
(38) using the pointer (35), or a combination of keys, to move through the mall As 
5 the shopper's position is changed, the presented image is updated accordingly. 

[0039] The image may be a photograph, such as images taken from within an actual 
mall, in the form of well-known graphic web objects (GIF, JPEG, etc.), or a simulation 
of an real view of the mall using vector and/or bitmap graphics similar to those used in 
gaming technologies, 

1 0 [0040] The images are provided with "hot spots" (62) or regions within the images 
that, when selected, activate a hyperlink to other images or other web pages. Such hot 
spots may be place over and around store doors, windows, and hallway tums. For 
example, to enter a particular store, the user could cUck on the door for the store. Or, 
to take a direct view into the display window of a store, the user could click on the 

1 5 image of the window. Methods to provide hot spots in graphic images with 
hyperlinks are well known in HTML and other common web object types. 
[0041] Turning to FIGURE 7, the logical process performed by the application 
program is shown. Upon initiaUzation (89) of the mall in the mobile device, the mall 
and store data is downloaded (802) via a communications network from a server 

20 (803). This may be a download over the Internet in the form of File Transfer Protocol 
(FTP), email attachement, or other suitable file and data download protocol. The data 
may include multiple graphic image files and audio recording files, each indexed to 
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positions and coordinates in one or more mall map files. In an enhanced embodiment 
of the invention, store price lists, product descriptions and product images may also be 
downloaded to enable off-line shopping and ordering. All of these data items are 
stored (804) in the mobile device's memory (84) for use by the application program, 
5 whether the device is connected online or is off-line during the subsequent shopping 
session. 

[0042] As described in the related apphcation, the mall floor plan or map is divided 
into discrete coordinates (81), and the shopper*s position is set to an initial position 

(82) to initiaUze the shopping session. 

1 0 [0043] During the shopping session, images are displayed and sounds are played 

(83) by the application program corresponding to the current position of the shopper 
in the cybermall floor plan. In the preferred embodiment, this is accomplished by 
querying the mobile device's memory for relevant image and sound web objects 
including JPEG and GIF image files and "wave" audio files. These data objects are 

15 retrieved (87) fi*om memory (84), and displayed and played on the mobile device's user 
interfaces. In an enhanced embodiment, these data objects may include streaming 
video and/or video clips. 

[0044] Further according to the preferred embodiment the mini-map display and 
compass or other directional indicator are updated (85) to reflect the current position 
20 within the mall. 

[0045] Each time a movement command is entered by the shopper, such as pressing 
an arrow key or dragging the position indicator, the movement command is received 
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(86) by the application program, which then updates the position coordinates of the 
shopper (88), and subsequently the images and sounds are changed (83) to correspond 
with the new position. The coordinate system used may be 2-dimensional, such as for 
a "one story" mall, or multi-dimensional to represent a multi-story mall. 
5 [0046] This process continues as the shopper "moves" through the mall, each move 
causing new images and/or sounds to be retrieved and played to the web browser user, 
until a hot spot or hyperlink is selected (87), at which time the application program is 
repointed to the linked address. This may link to a normal web page, such as an 
ordering or shopping cart page, or may link to another virtual shopping experience 
10 appHcation program section, such as entering a store through the "door" on the store's 
image. In this latter case, a new map, perhaps a detailed map of the interior of a store, 
may be provided along with presentation of images and sounds relevant to the 
shopper's position within the store. 

[0047] Also according to the preferred embodiment, each time a link is selected to 
1 5 another web page or to another virtual shopping session, a record is created of the 
shopper's last position in the current session, such as by storing a value in the mobile 
device's memory (84). This allows the appHcation program to remember the "exit 
point" from the current session, in order to allow the shopper to return from the next 
shopping visit or web page. In this case, when returning to a previous session, the 
20 process re-starts from point 2 (801) as shown in FIGURE 7, wherein the current 

position is retrieved from the previous position memory such as by retrieving a cookie 
from the browser. For example, if a shopper selects a hyperlink to a order entry web 
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page, he may complete the order and then return to the same position in the mall 
where he left. Or, he may enter a store map through a hot spot on the image of the 
front of the store, shop throughout a "sub-map'' of the interior of the store, and return 
back "through the front door" into the hallway of the mall by returning to his last 
5 position in the previous shopping session. 

[0048] This allows the invention to be used to "nest" visits to an infinite level, so 
that malls may be divided in to linked sections, such as upstairs and downstairs, stores 
can be linked to malls, and departments may be linked to stores. This modularization 
of the representation of the mall in the preferred embodiment allows the different mall 
10 sections, stores and departments to be developed and maintained as separate groups of 
web objects interrelated by the link definitions. It also lends itself very well to object 
oriented programming techniques for implementation. 

[0049] This position memorization and local storage of data objects allows the 
shopper to navigate the mall while connected online or while off-line. Off-line 
1 5 shopping capabihties allow the user to minimize the connection time consumed by the 
mobile device, which is usually a premium resource on a mobile device, measured in 
minutes of connection or units of data transferred. 

[0050] A further enhancement of the invention allows for the downloaded data 
objects to include store price lists, product descriptions, and product images. This 
20 may allow the user to add a particular item to a "shopping cart" (805) and to complete 
a purchasing process such as filling out an order form (85). This order form may then 
be transmitted during an online period by email, secure hypertext transfer (HTTPs), or 
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other data transfer protocol such as common gateway interface (CGI), using the 
mobile device's resident appHcation programs and library functions. These price lists 
and product descriptions may be provided with expiration dates such that old or stale 
data would not be used to place new orders, 

5 [0051] Using these coordinated, relevant presentations of information (sights, 
sounds, and maps), the user is given a sense of physical relationship of the stores 
within the mall, and a sense of the store environment while shopping in the store. For 
example, while in the "hallways" of an upscale cybermall, the user may hear classical 
or jazz music and see images of designer flooring, wall treatments, and planters. When 

10 entering a nature and science products store, the sounds change to new age music, and 
the images change to nature and science products displayed on shelves and counter 
tops. In another mall, perhaps an outlet mall, background audible announcements 
about specials in various stores may be heard while in the hallway, and when entering a 
clothing store specializing in outdoor products, nature sounds may be heard. 

1 5 [0052] While specific details of the preferred embodiment and alternate 

embodiments have been disclosed herein, it will be recognized by those skilled in the 
art that many substitutions, variations and alternate embodiments may be adopted 
without departing from the spirit and scope of the invention. For example, other 
programming techniques, device hardware platforms, and data object types may be 

20 adopted as suitable and as they become available. The scope of this invention should 
be limited only by the language of the following claims. 
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